There is now a wealth of data available online. These come from a variety of sources (crowdsourced data, online transaction data, administrative data, and so on), and in many formats (csv file, xml or json files, or through application program interfaces (APIs)). You will often want to access such data, and use it for your own work or research.
This tutorial will show you how to access data from the web using QGIS. We will write some python scripts (don’t worry, you will only have to copy and paste) and also use some plugins available in QGIS.
We will perform some web-scraping, for getting data from websites taking advantage of their HTML tags, and we will also be collecting tweets available through the twitter API.
This tutorial is meant to give you a flavour of how you can access data from online resources, and import it directly into QGIS. The code will be here for your reference, and this tutoarial will be available online here. If you have any questions later, do not hesitate to get in touch **reka.solymosi@manchester.ac.uk**
The main tool we will be using is QGIS, and its plugins. However we will also be making use of some Python code to use within the QGIS environment. Python is the language on which QGIS is built, and many of the plugins you might use will have been written by people with such code. It’s possible to write your own plugins, or to write scripts which you can automatically execute from within QGIS. Here I will be showing you the code directly for one exercise (web scraping) so that you can get a sense of what such a script would be doing, and how you can edit it to fit your needs.
For those potentially unfamiliar with it Python is … [PUT IN SOME HISTORY]
You can write such code in many dedicated development environments, however for our purposes here, QGIS provides a built-in console where you can execute python code. This console is a quick way to learn scripting and do quick data processing.
You can open the Python Console by going to Plugins > Python Console:
This will open a little window, where you can paste the Python code from this tutorial, or write your own. While no programming experience is required (or taught really) here, I will describe each bit of code that we use in detail, so that you have an understanding of what you are doing, why, and how you can change this if you wanted to implement it in your own work.
So to demistify this process for anyone who might not have written any code before, let’s carry out a quick exercise.